This post today isn’t huge news. It’s a well used technique, probably the most common tool used amongst Web Content Marketing. A/B Testing. It’s a rather simple method that compares 2 “variants” and takes a look at their performance.
The groups are split into two. Usually with the same length of test subjects for both (doesn’t always need to be). The two are identical, with the exception of one variation that might affect a behavioral change by the other user. For example, a website owner might want to test the effectivity of a stock image as the main image as opposed to a photo of the company to increase free trial rates. They would test the difference, reflecting a potential increase of the bottom line.
Some confuse it and see similarities with the ANOVA hypothesis testing and paired sample t-test, but we have to be very careful in distinguishing the differences.
1. We immediately notice that the ANOVA test is out of question, as it is a test that looks for differences in means of multiple groups. A/B only has 2. For that, taking a look into [multivariate testing](https://en.wikipedia.org/wiki/Multivariate_testing_in_marketing) will be in our benefit.
2. We are in a weird territory where product and the response from both populations are tested. However, the experimental unit (population of interest) really is the people (we can tell by where the deviation really comes from). Therefore, we should be closer in relating this method to [Welch's t-test](https://en.wikipedia.org/wiki/Welch%27s_t-test), which accounts for differences in population variance. Yet it is clear that we are not looking for differences in means. Let's just say it can be used as a tool within the A/B test to see if there is a statistical difference.
I have rarely seen this applied in situations other than user experience and web content marketing (though it easily can be). My assumption is that this is heavily used within the web sphere due to the ease of data collection. Economically, I would assume more utils per cost to retrieve data in that sense.
We will begin right away. As always, my general use tidyverse package may come in handy, but is not really necessary. We will need the chron package to separate our date and time variables as well.
library(tidyverse)
library(chron)
Say that we collected data on two types of website layouts. One looks like this:
The other like this: